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@ An improved graphical keyboard. 

® A system and method for a graphical keyboard 
that benefits from the expressive power and intuitive 
ease of use associated with pen strokes and ges- 
tures, yet does not require complex character-rec- 
ognition software. The graphical keyboard responds 
differently to different kinds of pen strokes. For ex- 
ample, lowercase "a" is entered by tapping a stylus 
on the "a" key of the graphical keyboard. Uppercase 
"A" is entered by an upward stroke initiated over the 
"a" key. Likewise, by stroking in other directions, a 
user can express other modifiers to the basic char- 
acter, such as "control", "alt", "command." etc. 
Also, strokes In distinct directions can be used to 
express space, backspace, delete, and return 
characters, for example. IVIultiple strokes can be 
applied in sequence to a single key to express 
multiple modifiers. Visual feedback of pen strokes 
can be provided in the form of marks made by the 
pen. Pop-up menus can be used to provide help 
with and to facilitate memorization of command 
strokes and stroke sequences. The graphical key- 
board thus lets the expert user proceed by "feel," 



and prompts the novice user when and as neces- 
sary. 
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COPYRIGHT NOTIFICATION 

A portion of the disclosure of this patent docu- 
ment contains material which is subject to copy- 
right protection. The copyright owners have no 5 
objection to the facsimile reproduction, by anyone, 
of the patent document or the patent disclosure, as 
it appears in the Patent and Trademark Office 
patent file or records, but otherwise reserve all 
copyright rights whatsoever. io 

SOFTWARE APPENDIX 

An appendix of 5 pages is Included as part of 
this application. The appendix provides a source is 
code software program for implementation of an 
embodiment of the method of the Invention on a 
digital computer. The source code includes a sin- 
gle file written in the HyperCard scripting language. 
The code runs under the HyperCard program (ver* 20 
sion 2.0 or higher) for Apple Macintosh computers 
(Macintosh II or better hardware running System 
7.0 or higher operating software), well known to 
those of skill in the art. 

The source code represents unpublished work 25 
that is Copyright ©1993 Xerox Corporation. All 
rights reserved. Copyright protection claimed in- 
cludes all forms and matters of copyrightable ma- 
terial and information now allowed by statutory or 
judicial law or hereafter granted, including without 30 
limitation, material generated from the software 
programs which are displayed on the screen such 
as icons, screen display looks, etc. 

Although a particular software implementation 
Is provided in the Appendix, it will be apparent to 36 
those of skill in the art that a wide variety of 
programming languages and hardware configura- 
tions can readily be used based on this disclosure 
without departing from the scope of the invention. 

40 

BACKGROUND OF THE INVENTION 

The present invention relates to graphical user 
interfaces and more particularly to graphical user 
Interfaces for pen-based computing. 45 

Pen-based computing Is becoming Increasingly 
popular. The user of a pen-based computer inputs 
commands and data Into the computer using a pen 
or stylus, typically by printing or writing characters 
with the pen or stylus on the computer's display 50 
screen In a manner analogous to printing or writing 
with a pen on ordinary paper. Certain commands 
can be entered as pen strokes (gestures): for ex- 
ample, a word may be deleted by crossing it out. 
or a file may be opened by writing an "o" on Its 55 
representative icon. Examples of pen-based com- 
puting are found in systems ranging from palm- 
sized personal digital assistants, through slate-siz- 



ed machines isuch as notepad computers, and up 
to wall-sized interactive displays. Also, pen-based 
interfaces can sometimes be added or retrofitted to 
computer systems not originally designed to be 
pen-based, such as certain personal computer sys- 
tems. 

The usefulness of pen-based computing is 
limited because of the limited capabilities of char- 
acter recognition software. In many pen-based sys- 
tems, the user cannot write in cursive handwriting, 
but must print. This slows down input. Furthermore, 
the system has difficulty in recognizing printed 
characters. Consequently, the user must print care- 
fully (viz., slowly) and often must print in special 
windows and within designated boxes, and the sys- 
tem can require special training for each Individual 
user. Even so, character recognition performance 
can be relatively poor. In addition, pen-based sys- 
tems that use character recognition do not adapt 
well to run on conventional computer workstations 
equipped with a mouse, for example. 

Because of the limitations of the character- 
recognition approach to pen-based input, an alter- 
native approach is sometimes used. In this ap- 
proach, a graphical keyboard, visually analogous to 
the conventional mechanical (typewriter) keyboard 
commonly used for input in non-pen-based com- 
puter systems, appears on the computer display. 
The user enters text by using the stylus to tap on 
desired keys of the graphical keyboard. 

The graphical-keyboard approach offers certain 
advantages over the character-recognition ap- 
proach. Entering data through a graphical keyboard 
can be faster than printing (especially with numeric 
keypads), can be less susceptible to errors when 
properly implemented (i.e.. when the graphical 
keys are not too small), and does not require the 
system to be specially trained for each Individual 
user. Moreover, the approach is simple to imple- 
ment and integrate into existing software applica- 
tions. 

Unfortunately, known graphical keyboards suf- 
fer from numerous drawbacks. Although the graphi- 
cal keyboard is visually analogous to a mechanical 
keyboard, it is functionally far more limited. With a 
graphical keyboard, a user cannot "touch type" 
using all fingers of both hands, but instead must 
"hunt and peck" with the pen or stylus, as if typing 
with one finger on a mechanical keyboard. Typing 
speed on the graphical keyboard is constrained by 
the time required for the user to move the fin- 
ger/stylus from key to key on the keyboard. Also, 
the user cannot tap multiple keys simultaneously. 
In particular, the user cannot tap a modifier key, 
such as the "shift." "ctrl," "option," or "alt" key, at 
the sanhe time as an ordinary key, such as a letter 
or number key, but instead must first tap the modi- 
fier key and then tap the letter or number key. In 
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some systems the user must tap the modifier key 
once to set the modifier, tap the letter or number 
l<ey, and then tap the modifier key again to release 
the modifier (for example, tap "shift," "a," "shift" to 
input an uppercase "A"). This can be especially 
awkward when multiple modifiers must be com- 
posed to form a single character (for example, a 
doubly modified character such as ctrl-alt-x or a 
triply modified character such as ctrl-alt-X). 

Another problem with the graphical keyboard 
approach is that it places a heavy visual load on 
the user. Whereas with a mechanical keyboard, the 
user can memorize commonly used sequences of 
keystrokes and perform them "by feel" without 
looking at the keyboard, with the graphical key- 
board the user typically must look at the display 
before each tap of the stylus to ensure that the 
stylus is correctly placed. Furthermore, the graphi- 
cal keyboard tends to take up a large portion of the 
screen display, in part because the keyboard con- 
tains many or all of the modifier and function keys 
found in its mechanical counterpart. 

Because of its drawbacks, the graphical-key- 
board approach to pen-based input is often consid- 
ered cumbersome, and is used in pen-based com- 
puting systems only as a fallback when character 
recognition fails or is insufficiently reliable. The 
character-recognition and graphical-keyboard ap- 
proaches to pen-based computing are typically 
seen as mutually exclusive alternatives, not as 
complements. In particular, it has not been possible 
to combine the intuitive ease of use promised (if 
not always afforded) by the character-recognition 
approach with the speed, robustness and ease of 
implementation offered by the graphical keyboard 
approach. 

SUMMARY OF THE INVENTION 

The present Invention is a graphical keyboard 
that permits the user to do more than simple key- 
tapping. It lets the expert user proceed by "feel." 
and prompts the novice user when and as neces- 
sary. The graphical keyboard benefits from the 
expressive power and intuitive ease of use asso- 
ciated with pen strokes and gestures, yet does not 
require complex character-recognition software. 

The graphical keyboard of the invention re- 
sponds differently to different kinds of pen strokes. 
For example, lowercase "a" is entered by tapping 
a stylus on the "a" key of the graphical keyboard. 
Uppercase "A" is entered by an upward stroke 
initiated over the "a" key. Likewise, by stroking In 
other directions, a user can express other modifiers 
to the basic character, such as "control", "alt", 
"command," etc. Also, strokes in distinct directions 
can be used to express space, backspace, de- 
lete, and return characters, for example. Multiple 



strokes can be applied In sequence to a single key 
to express multiple modifiers. Visual feedback of 
pen strokes can be provided in the form of marks 
miade by the pen. Pop-up menus can be used to 
6 provide help with and to facilitate memorization of 
command strokes and stroke sequences. 

In another aspect, the present invention pro- 
vides an improved method for entering characters 
into a pen-based computer using a graphical key- 
70 board. 

A further understanding of the nature and ad- 
vantages of the invention will become apparent by 
reference to the remaining portions of the speci- 
fication and drawings. 

IS 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 illustrates a sample graphical keyboard; 
Fig. 2 schematically dieprcts how a lowercase 

20 character can be entered by tapping a key of 
the graphical keyboard of the invention; 
Fig. 3 schematically depicts how a uppercase 
character can be entered by an upward stroke 
on a key of the graphical keyboard of the Inven- 

25 tion; 

Fig. 4 schematically depicts how each modifier 
can be distinguished by the direction of Its par- 
ticular stroke; 

Fig. 5 schematically depicts an example of how 
30 modifiers are combined; 

Fig. 6 schematically depicts an example of how 
certain frequently used keys can be represented 

as diagonal strokes; 

Fig. 7 is a series of views in which a user enters 
35 a lowercase character by tapping a key of the 
graphical keyboard; 

Fig. 8 is a series of views in which a user enters 
an uppercase character by making an upward 
stroke on a key of the graphical keyboard; 

40 Fig. 9 is a series of views in which a user enters 
a control character by making a rightward stroke 
on a key of the graphical keyboard; 
Fig. 10 is a series of views in which a user 
enters a control uppercase character by making 

45 a compound rightward-upward stroke on a key 
of the graphical keyboard; 
Fig. 11 depicts the two modes of selecting from 
a marking menu; 

Fig. 12 is example of a radial menu hierarchy 
50 and the marks that select from it; 

Fig. 13 is an example of mark confirmation; 
Fig. 14 schematically depicts a pop-up menu 
revealing a stroke vocabulary of eight stroke 
directions; 

56 Fig. 15 is a series of views in which a user is 
presented with and selects from a marking 
menu; 
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Fig. 16 is a series of views showing an exannple 
of marl<ing menus used to combine modifiers in 
the context of the present invention; 
Fig. 17 is a series of views showing how an 
eight-way marking menu can be used with the 
graphical keyboard of the invention; 
Fig, 18 illustrates a pen-based system suitable 
for implementing the invention; 
Fig. 19 illustrates an alternative pen-based sys- 
tem in which the display screen and input sen- 
sor are decoupled; 

Fig. 19A illustrates an input sensor with an af- 
fixed keyboard image; 

Fig. 20 illustrates a mouse-based system suit- 
able for implementing the invention; 
Fig. 21 is a flowchart of steps for responding to 
an event according to an embodiment of the 
invention that incorporates marking menus; 
Fig. 22 is a flowchart of steps for processing a 
Pen Down event; 

Fig. 23 is a flowchart of steps for processing a 
Pen Drag event; 

Fig. 24 is a flowchart of steps for processing a 
PenStill event; 

Fig. 25 is a flowchart of steps for processing a 
PenUp event; 

Fig. 26 is a flowchart of steps for determining 

the modifiers corresponding to a mark; 

Fig. 27 is a flowchart of steps for determining 

points of articulation along a mark; and 

Fig. 28 is a flowchart of steps for matching 

points of articulation along a mark with centers 

of corresponding menus. 

DESCRIPTION OF SPECIFIC EMBODIMENTS 

1. Introduction 

The present invention provides a system and 
method for an improved graphical keyboard that 
responds differently to different kinds of pen 
strokes. By applying appropriate strokes, the user 
can easily type characters that on mechanical key- 
boards are typed by pressing multiple keys si- 
multaneously. Also, the user can type certain fre- 
quently used keys, such as space, backspace, 
delete, and return, from anywhere on the key- 
board. Using pen strokes makes typing easier and 
faster, reduces the visual toad on the user, and 
makes It possible for the keyboard to take up less 
space in the screen display. 

2. Examples 

To understand the invention, it is helpful to 
begin with some illustrative examples of its use, as 
depicted in Figs. 1-10. A displayed image of a 
sample graphical keyboard according to the inven- 



tion is shown In Fig. 1 , Tapping on any of the keys 
with the pen or stylus has the same effect as 
pushing the analogous key on a mechanical key- 
board. Thus to enter a lowercase character, a digit. 

5 or another unmodified character such as a period 
or comma, the user simply taps the desired key. 

To enter an uppercase character, rather than 
tapping, the user makes an upward stroke, initiated 
on the desired key. For example, as Illustrated In 

10 Fig. 2, tapping on the "a" key results in "a." An 
upward stroke on "a" results in "A" (Fig. 3). 

The user accesses other modifier keys, such 
as control, alternate, option, or command, in a 
similar manner. Each modifier is distinguished by 

75 the direction of Its particular stroke. An example 
mapping is shown in Fig. 4. According to the 
mapping of Rg. 4, a stroke to the right indicates 
the control ("Ctrl") modifier. Thus a rightward pen 
stroke Initiated over the character "c" causes the 

20 graphical keyboard to output the character "ctrl-c." 
A stroke to the left indicates the option modifier, 
and a downward stroke indicates the command 
modifier- 
Modifiers can easily be combined. For exam- 

25 pie, once again applying the mapping of Fig. 4. a 
stroke to the right starting on the character "c." 
followed by an upward stroke expresses "ctrl-C." 
as distinct from "ctrl-c." This is illustrated In Fig. 5. 
Additional stroke directions can be used to 

30 substitute for certain frequently used keys. For 
example, pen strokes in the diagonal directions can 
be used to express space, backspace, delete, 
and return. A possible layout of these commands 
is shown in Fig. 6. This class of strokes is distinct 

35 from the modifier strokes discussed above in con- 
nection with Figs. 3, 4, and 5, in that the meaning 
of these strokes is not affected by where the stroke 
is made. Thus a diagonal stroke up and to the left 
is interpreted in the mapping of Fig. 6 as a back- 

40 space regardless of whether it is made on the "a" 
key, the "c" key, or any other key of the graphical 
keyboard. 

Visual feedback can assist the user In applying 
pen strokes to the graphical keyboard. As the user 

45 moves the pen across the screen, a line appears 
on the screen to show where the pen has been, as 
though the pen contained ink that left a mark on 
the screen. Once the pen stroke is recognized, the 
character is entered and the "ink trail" disappears. 

50 This is illustrated, for example, in Fig. 8. 

Each of Figs. 7-10 is a series of views showing 
how a user enters Input into the graphical keyboard 
of the Invention. The series of two views of Fig. 7 
shows a user entering a lowercase character using 

55 the graphical keyboard of the invention. The user 
does this by tapping the desired key of the graphi- 
cal keyboard with the pen or stylus. More pre- 
cisely, the user taps the pen against the surface of 
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the screen on which the keyboard image of the 
graphical keyboard is displayed, with the point of 
the pen striking the screen surface in proximity to 
the location of the desired key within the keyboard 
image. 

In the first view of Fig. 7, the user places the 
pen monnentarily on the key corresponding to the 
letter or other character to be entered (an "a" in 
this example). The key is highlighted in the display. 
The character buffer, which is represented by the 
small rectangle next to the view of the keyboard, 
pen, and user, is empty at this stage. In the second 
view, the user removes the pen from the graphical 
keyboard, thereby completing the tapping motion. 
The graphical keyboard produces the character "a" 
and sends it to the character buffer. The highlight- 
ing disappears from the display. 

The series of three views of Fig. 8 shows a 
user entering an uppercase character using the 
graphical keyboard of the invention. The user does 
this by making an upward stroke or mark on the 
desired key of the graphical keyboard. 

In the first view of Fig. 8, the user places the 
pen or stylus on the key corresponding to the 
character to be entered (an "A" in this example). 
The key is highlighted in the display. The character 
buffer is empty. 

In the second view of Fig. 8, the user moves 
the pen "upward"-that is. toward the topmost row 
of the keyboard-while continuing to hold the pen 
against the keyboard. The user maintains contact 
between the pen point and the surface of the 
screen on which the keyboard image is displayed 
substantially throughout the upward motion. This is 
called dragging the pen. The display shows a line 
or mark (an Ink trail) where the user has dragged 
the pen. The character buffer remains empty at this 
stage, because the character to be entered has not 
yet been recognized. The key remains highlighted 
in the display. 

In the third view of Fig. 8, the user removes the 
pen from the keyboard surface, completing the 
upward stroke. The line or mark disappears, as 
does the highlighting, and the character produced 
by the graphical keyboard in response to the up- 
ward stroke appears in the character buffer. 

The series of three views of Fig. 9 shows a 
user entering a control character using the graphi- 
cal keyboard of the invention. A "control character" 
is produced on a conventional, mechanical key- 
board by holding down the modifier key marked 
"control" or "Ctrl" while pressing a character key. 
Pressing the "Ctrl" key while striking the key for 
the letter "a," for example, produces the character 
called "control-a," which can also be written as 
"ctrl-a" or "^a." On the graphical keyboard of the 
Invention, the user can enter a control character by 
making a rightward stroke or mark on the desired 



key of the graphical keyboard. 

In the first view of Fig. 9, the user places the 
pen or stylus on the key corresponding to the 
character to be entered (the key "a" corresponding 
6 to the character "'^a" In this example). The key is 
highlighted iri display. The character buffer is emp- 
ty. 

In the second view of Fig. 9. the user drags the 
pen to the right: The user moves the pen to the 

10 right while continuing to hold the pen against the 
keyboard, and maintains contact between the pen 
point and the surface of the screen on which the 
keyboard image is displayed substantially through- 
out the rightward motion. The display shows a line 

76 or mark where the user has dragged the pen. The 
key remains highlighted. The character buffer re- 
mains empty at this stage, because the character 
to be entered has not yet been recognized. 

In the third view of Fig. 9. the user removes the 

20 pen from the keyboard surface, completing the 
rightward stroke. The line or mark disappears, as 
does the highlighting of the key, and the character 
produced by the graphical keyboard in response to 
the rightward stroke appears in the character buff- 

25 er. 

Fig. 10 shows that modifiers can be combined 
by making compound strokes that combine strokes 
of different directions. The series of three views of 
Fig. 10 shows a user entering a control uppercase 

30 character using the graphical keyboard of the in- 
vention. A "control uppercase character" is pro- 
duced on a conventional, mechanical keyboard by 
holding down the "shift" and "ctri" modifier keys 
simultaneously while pressing a character key. 

35 Pressing the "shift" and "ctri" keys while striking 
the key for the letter "a," for example, produces 
the character called "control-shift-a," which can 
also be written as "ctrl-A" or "'"A." On the graphi- 
cal keyboard of the invention, the user can enter a 

40 control uppercase character by making compound 
rightward-upward stroke or mark on the desired 
key of the graphical keyboard. 

In the first view of Fig. 10, the user places the 
pen or stylus on the key corresponding to the 

45 character to be entered (the key "a" corresponding 
to the character "^A" in this example). The key is 
highlighted in the display. The character buffer is 
empty. 

In the second view of Fig. 10, the user drags 
50 the pen a distance to the right and then a distance 
"upward "-that is. toward the topmost row of the 
keyboard. The user maintains contact between the 
pen point and the surface of the screen on which 
the keyboard Image is displayed substantially 
65 throughout the rightward-and-then-upward motion. 
The display shows a line or mark where the user 
has dragged the pen. The key remains highlighted 
in the display. The character buffer remains empty 
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at this stage, because the character to be entered 
has not yet been recognized. 

In the third view of Fig. 10. the user removes 
the pen from the keyboard surface, completing the 
compound stroke. The line or mark disappears, as 6 
does the key highlighting, and the character pro- 
duced by the graphical keyboard In response to 
the stroke appears in the character buffer. 

Although the example of Fig. 10 shows the 
rightward portion of the compound stroke preced- io 
Ing the upward portion, the same "^A" character 
Is produced if the upward portion of the stroke is 
performed before the rightward portion. Other kinds 
of modifiers can be combined by making com- 
pound marks in other directions. For example, if a 15 
leftward stroke indicates the "alt" modifier, then a 
leftward-upward or upward-leftward compound 
stroke combines the "alt" and "shift" (uppercase) 
modifiers, and a leftward-rightward or rightward- 
leftward compound stroke combines the "ctrl" and 20 
"alt" modifiers. 

3. Marking Menus 

An interaction technique known as marking 25 
menus is used to Implement the presentation of 
and response to pen strokes in the graphical key- 
board of the present invention in the specific em- 
bodiment to be described. This section presents a 
general description of the technique; section 4 de- 30 
scribes the technique more specifically as it ap- 
plies to the present invention. A complete descrip- 
tion of the marking menus technique can be found 
in Gordon P. Kurtenbach, "The Design and Evalu- 
ation of Marking Menus." Ph. D. dissertation. Dept. 35 
of Computer Science, Univ. of Toronto, 1993, which 
is incorporated herein by reference as though set 
out in full. 

3.1 Mark and Related Concepts 40 

The marking menus technique is intended to 
support the novice user and expert user alike, and 
to provide an efficient mechanism to support the 
user's transition from novice to expert. In marking 4S 
menus, as a user "writes" on the computer display 
surface with a pen or stylus the computer simu- 
lates the appearance of ink by leaving behind a 
trail (commonly refen^ed to as an ink- trail) of 
pixels whose color or shade contrasts with that of so 
the background pixels. 

A mark is the series of pixels that are changed 
to the special "ink" color or shade when the pen is 
pressed and then moved across the display. The 
pixels that are changed to the ink color are those 55 
which lie directly under the tip of the pen as it is 
moved across the display. Freehand drawings, 
ranging from meaningless scribbles to meaningful 
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line drawings and symbols, including handwriting, 
are examples of marks. The act of drawing a mark 
is referred to as marking. 

Marks can be created not only with a pen but 
also with other types of input devices. For example, 
a mouse can leave a trail of ink behind the tracking 
symbol when the mouse button is pressed and the 
mouse is dragged. Some systems use a pen and 
tablet. In this case, marks are made on the display 
by writing on the tablet instead of the display. Still 
other systems use a trackball. 

Marking menus find application in pen-based 
computing systems and interfaces that can inter- 
pret as commands and parameters the various 
marks a user makes. (This is in contrast with sys- 
tems in which a user's marks cannot be recognized 
by the system and serve strictly as annotations for 
the user's benefit) 

Much of the literature on pen-based computing 
refers to marks as gestures. However, the term 
gesture is inappropriate in the context of marking 
menus. Although it is true that creating a mark 
does involve a physical gesture, the thing that is 
interpreted by the system is the mark itself. For 
example, the "X" mark requires a completely dif- 
ferent physical gesture if performed with a pen 
instead of a mouse. Gesture is an important aspect 
of marking because some marks may require awk- 
ward physical gestures with the input device. How- 
ever, the two terms can be distinguished. The term 
mark is more appropriate for pen-based computer 
systems or applications that emulate paper and 
pen. and will be used hereinbelow. 

3.2 Why Use Marks? 

Conventional human-computer interfaces are 
asymmetric in terms of input and output capabil- 
ities. There are number of computer output modes: 
visual, audio and tactile. Most computers exten- 
sively utilize the visual mode; high-resolution im- 
ages that use thousands of colors of can be dis- 
played quickly and in meaningful ways to a user. In 
contrast, a computer's ability to sense user input is 
limited. Humans have a wide range of communica- 
tion skills such as speech and touch, but most 
computers sense only a small subset of these. For 
example, keyboards only sense finger presses (but 
not pressure) and mice only sense very simple arm 
or wrist movements. Therefore, the advent of the v 
pen as a computer input device provides the op- 
portunity to increase input bandwidth through the 
use of marks. 

A major motivation for using marks Is the in- 
adequacy of mouse and keyboard interfaces. This 
is exemplified by applications that are controlled 
through button presses and position information. 
(The term buttons is used here as a generic way 
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of describing menus Items, dialog box Items. Icons, 
keys on a keyboard, etc., which are typical of 
direct manipulation interfaces.) Expert users of 
these types of systems find the interfaces inad- 
equate because button interfaces are inefficient. 
The existence of interaction techniques that over- 
ride buttons for the sake of efficiency is evidence 
of this. Experts, having great familiarity with the 
interface, are aware of the set of available com- 
mands. They no longer need menus to remind 
them of available commands, and they find that 
invoking commands through menu displays be- 
comes very tedious. 

Designers have addressed this problem in sev- 
eral ways. One solution is accelerator keys which 
allow experts direct access to commands. An ac- 
celerator key is a key on the keyboard which, when 
pressed, immediately executes a function asso- 
ciated with a menu item or button. The intention is 
that using an accelerator key saves the expert user 
the time required to display and select a menu 
item or button. Many systems display the names of 
accelerator keys next to menu items or buttons to 
help users learn and recall the associations be- 
tween accelerator keys and functions. 

Accelerator keys have their problems. Arbitrary 
mappings between accelerator keys and functions 
can be hard to learn and remember. Sometimes 
mnemonics can be established between accelera- 
tor key and function (e.g.. control-o for "open"), but 
mnemonics quickly run out as the number of accel- 
erator keys increases. Further confusion can be 
caused by different applications using a common 
key for different functions or by different applica- 
tions using different accelerator keys for a common 
function. Experts must then remember arbitrary or 
complex mappings between keys and functions 
depending on application. 

Another way of supporting the expert user is 
by supplying a command- fine interface in addi- 
tion to a direct manipulation interface. However, 
command-line interfaces also have problems, nota- 
bly because they are radically different from direct 
manipulation interfaces. To become an expert, a 
novice must learn another entirely different inter- 
face. 

Marks overcome the problems of accelerator 
key and command-line interface solutions. Marks, 
because of their symbolic nature, can make func- 
tions more immediately accessible. Rather than 
triggering a function by a button press, a mark can 
signal a command. For example, a symbolic mark 
can be associated with a function and a user can 
invoke the function by drawing the symbol. In the- 
ory, because marks can be used to draw any 
symbol or series of symbols, marks can provide a 
quicker method of choosing a command than 
searching for a physical or graphical button and 



■ pressing it. In practice, the number of marks Is 
limited by the system's ability to recognize sym- 
bols and a human's ability to remember the set of 
symbols. Nevertheless, even if only a small set of 

5 marks are used, a user can Invoke the associated 
functions immediately. 

Marks can also be used to hide functions. 
Functions that are not provided by means of but- 
tons can nonetheless be provided by marks. This 

10 makes a full-functioned interface possible with a 
smaller number of buttons, which in turn means a 
more efficient use of available display area. In 
many case:, moreover, a mark can be much more 
effective than a button for invoking a function. 

76 An additional motivation for using marks is that 

there are some intrinsic qualities that marks have 
which can provide a more "natural" way to express 
otherwise difficult or awkward concepts (such as 
spatial or temporal information). In particular, marks 

20 provide a straightforward way to embed multiple 
command attributes into a single mark. 

3.3 Marking Menus and Related Concepts 

25 A marking menu is an interaction technique 

that allows a user to select from a menu of items. 
There are two ways, or modes, in which a selection 
can be p)erformed: 

(1) Menu mode. In this mode a user makes a 

30 selection by displaying a menu. A user enters 

this mode by pressing the pen against the dis- 
play and waiting for a short time interval, for 
example, approximately 1/3 of a second. This 
action is sometimes referred to as press- and- 

35 wait. A radiai menu (or pie menu) of items is 
then displayed centered around the pen tip. A 
radial menu is a menu wherein the menu Items 
are positioned in a circle surrounding the cursor 
and each item is associated with a certain sector 

40 of the circle. A user can select a menu Item by 
moving the pen tip into the sector of the desired 
item. The selected item Is highlighted and the 
selection is confirmed when the pen is lifted 
from the display. At the center of the radial 

45 menu is a center circle, typically, a circle of 
about 1 cm in diameter, highlighted or othenvise 
visually indicated. The pen can be moved within 
the center circle without causing a menu selec- 
tion. 

50 (2) Mark mode, in this mode, a user makes a 
selection by drawing a mark. A user enters this 
mode by pressing the pen against the display 
and immediately moving in the direction of the 
desired menu item. Rather than displaying a 

56 menu, the system draws an ink-trail following 
the pen tip. When the pen is lifted, the item that 
corresponds to the direction of movement is 
selected. 
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Fig. 11 depicts two views that illustrate, respec- 
tively, the two mode: of selecting a command from 
a marking menu. The first view at the left of Fig. 1 1 
shows a user selecting a menu choice in menu 
mode. In this example, two menus are presented in 
succession to the user. The user choose: the 
"paste" menu choice by dragging the pen into the 
upper right-hand quadrant of the first menu, and 
then, without lifting the pen off the screen, chooses 
the "clipboard c" menu choice by dragging the 
pen into the bottom quadrant of the second menu. 
The effect of this series of choices is to cause data 
to be pasted (inserted into a working document) 
from a clipboard (storage buffer) called Clipboard 
C. Although the first and second menus are shown 
for clarity of exposition as appearing simultaneous- 
ly on the display screen in Fig. 11, in some em- 
bodiments-and in particular, in the embodiments 
of the graphical keyboard of the invention de- 
scribed herein-the first menu disappears when the 
second menu appears, so that the user is pre- 
sented with only one menu at a time. 

The second view at the right of Fig. 11 shows a 
user selecting the same menu choice as in the first 
view, but this time doing so in mark mode. The 
user makes substantially the same pen motions 
and thus the substantially same mark (a diagonal 
upward-rightward line segment followed by a down- 
ward line segment) as in the first view, but the 
radial menus do not appear. The effect of this 
series of chbices is once again to cause data to be 
pasted from a clipboard called Clipboard C. The 
second view of Fig. 11 illustrates an important 
concept of marking menus: The physical move- 
ment required to select an item using a mark in 
mark mode mimics the physical movement in- 
volved in selecting an item in menu mode. 

Marking menus can be hierarchical. In menu 
mode, if a menu item has a subitem associated 
with it, rather than lifting the pen to select the item, 
the user waits with the pen pressed down to trigger 
the display of the submenu. The submenu is also a 
radial menu. The user can then select an item from 
a submenu in the manner previously described. In 
mark mode, a user makes a selection by drawing a 
mark where changes in direction correspond to 
selections from submenus. The example of Fig. 11 
is an example of selecting from hierarchical menus 
using menu mode and mark mode. The first menu 
is at the top of the hierarchy; the second menu is a 
submenu of the first and Is beneath the first in the 
hierarchy. 

Using radial menus In this way. produces a set 
of mark which consist of a series of line segments 
at various angles ("zig-zag" marks). Marking 
menus which have no hierarchic items produce 
strictly straight line segments. Rg. 12 shows an 
example of a menu hierarchy and its associated set 



of marks. The left-hand view of Fig. 12 shows the 
menu hierarchy. The top-level menu in Fig. 12 is a 
numeric menu containing the four menu choices 1, 
2, 3, and 4. Each of these choices has associated 

5 with it an alphabetic submenu consisting of the 
choice: a, b. c, and d. The right-hand view of Fig. 
12 shows the mark set. A mark's label indicates the 
menu items It selects. A dot indicates the starting 
point of a mark. 

10 It is also possible to verify the items associated 

with a mark or a portion of a mark. This is some- 
times referred to as mark confirmation. In this 
case a user draws a mark but presses-and-waits at 
the end of drawing the mark. The system then 

75 displays radial menus along the mark "as if** the 
selection were being performed in menu mode. 
Rg. 13 shows an example of this in a menu with 
three levels of hierarchy. In the left-hand view of 
Fig. 13, the user draws the first part of the mark. 

20 then waits with the pen pressed down for the 
isystem to recognize the selection so far. In the 
right-hand view of Fig. 13, the system then dis- 
plays its interpretation of the mark, and goes into 
menu mode for the completion of the selection. 

25 Other types of behavior can occur when select- 

ing from a marking menu, such as backing-up in a 
menu hierarchy or reselecting an item in menu 
mode. 

30 3.4 Summary 

The following is a summary of the design fea- 
tures that make marking menus a valuable and 
unique interaction technique. Marking menus: 
35 # Allow menu selection acceleration without a 

keyboard. 

• Permit acceleration on all menu item:. 

• Minimize the difference between the menu 
selection and accelerated selection. 

40 • Permit pointing and menu selection accelera- 
tion with the same input device. 

• Utilize marks that are easy and fast to draw. 

• Use a spatial method for learning and re- 
membering the association between menu 

45 items and marks. 

• Are Implementable as a "plug-in" software 
module. 

With conventional menus, the user moves a 
cursor (e.g., using a mouse) to a particular absolute 

50 or relative screen location in order to select an item 
from the menu. With marking menus, It is the 
direction of motion rather than the particular loca- 
tion moved to that causes menu selection. For 
example, moving the cursor upwards on the screen 

55 can select a first menu item; down, a second; right, 
a third; left, a fourth. Additional selections can be 
associated with diagonal cursor motions. Marking 
menus are arranged radially (as "pie menus") rath- 
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er than linearly, so that each direction of motion Is 
associated with a unique menu selection. 

The geonnetrlcal nature of marking menus 
makes them easy for users to memorize. A user 
learns and remembers spatial pathways through 
the menus. Such pathways can be easier to re- 
member than combinations of accelerator keys that 
are conventionally used to substitute for menu se- 
lections. Whereas a sequence of accelerator keys 
corresponding to a sequence of selections from 
multiple levels of hierarchical menus can be some- 
what arbitrary, the path traced by the pen In skip- 
ping over intermediate marking menus is the same 
that is used to select them when the menus are 
displayed. In short, marking menu: show the menus 
to the novice user, and leave the expert user just 
making the marks. 

4. Marking Menus in the Present Invention 

As applied in the present invention, the mark- 
ing menus technique makes menus available to 
assist the user who has not memorized the pen 
strokes, and hides menus from the user who does 
not require them. At any time, the user can activate 
a radial pop-up menu which show: the available 
choices by holding the pen down without moving it 
laterally. The user can then make the desired se- 
lection by moving the pen (that is. the point or tip 
of the pen, which is in contact with the Input- 
sensing display screen or input tablet) into the 
"slice" of the pie occupied by the desired menu 
choice. Simply raising the pen off the display 
screen without lateral movement causes the menu 
to disappear and no selection to be made. 

Figs. 14-17 depict illustrative examples of 
marking menus as applied in the context of the 
present invention. An example of a radial or pie 
menu corresponding to the eight strokes defined in 
the example: of Figs. 4 and 6 is shown in Rg. 14. 
The directions of the strokes substantially corre- 
spond to the slices of the pie. 

Fig. 15 is a series of views in which a user 
enters a " ^ a" character using the graphical key- 
board of the invention, with marking menus used in 
menu mode. 

In the first view of Fig. 15, the user presses 
and holds the pen on the "a" key of the graphical 
keyboard. More precisely, the user presses and 
holds the pen against the display screen at the 
location of the "a" key of the keyboard image of 
the graphical keyboard. The key is highlighted In 
the display. Thereafter, a radial menu showing four 
choices of modifier-"uppercase," "option," "con- 
trol." and "command"-appear5 on the screen, 
overlaid on the keyboard image and approximately 
centered around the location of the tip of the pen. 
Other modifiers or other menu choices can be 



provided in alternative embodiments. Also, at this 
point, the user can simply lift the pen off the 
screen to cancel the menu display, in which case 
no character is generated by the graphical key- 
5 board. 

In the second view of Fig. 15. the user moves 
the pen to the right, maintaining the pen in contact 
with the screen, to select the "control" modifier. 
The radial menu remains displayed. The key re- 

10 mains highlighted. 

In the third view of Fig. 15, the user lifts the 
pen point off the screen. This causes the "control" 
modifier to be applied to the "a" key, so that the 
graphical keyboard generate: the "^a" character. 

76 The radial menu disappears, as does the key high- 
lighting, leaving the keyboard image as it originally 
appeared. 

Fig. 16 is a series of views in which a user 
enters a " ^ A" character using the graphical key- 

20 board of the invention, with marking menus used in 
menu mode. 

In the first view of Fig, 16, the user presses 
and holds the pen on the "a" key of the graphical 
keyboard. More precisely, the user presses and 

25 holds the pen against the display screen at the 
location of the "a" key of keyboard image of the 
graphical keyboard. Thereafter, a radial menu 
showing four choices of modifier~"uppercase." 
"option." "control," and "command "-appears on 

30 the screen, overlaid on the keyboard image and 
approximately centered around the location of the 
tip of the pen. 

In the second view of Fig. 16, the user moves 
the pen to the right, maintaining the pen in contact 

35 with the screen, to select the "control" modifier. 
The selected menu choice (the "control" modifier) 
is highlighted. The radial menu remains displayed 
and the key remains highlighted. 

In the third view of Fig. 16, the user holds the 

40 pen still, maintaining the pen at the location pre- 
viously reached upon selection of the "control" 
modifier. Thereafter, the first menu of four choices 
disappears and is replaced by a second radial 
menu which shows the remaining available choices 

45 of modifier available to be applied to this character- 
-"uppercase," "option," and "command." This sec- 
ond menu appears on the screen overlaid on the 
keyboard image and approximately centered ar- 
ound the location of the tip of the pen. The key 

50 remains highlighted, although it is partially ob- 
scured by the menu in this example. 

In the fourth view of Fig. 16. the user moves 
the pen upwards, that is, toward the top of the 
screen display, to select the "uppercase" modifier. 

65 The selected menu choice is highlighted. The ra- 
dial menu remains displayed, and the key remains 
highlighted. 
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In the fifth view of Fig. 16. the user lifts the pen 
off the screen. This causes the "control" and "up- 
percase" modifiers to be applied to the "a" key, so 
that the graphical keyboard generates the -^A" 
character. The radial menu disappears, as does the 
key highlighting, leaving the keyboard image as it 
originally appeared. 

Menus can be nested to levels deeper than the 
two levels shown In Fig. 16. For example, three 
levels of modifier can be applied to a character, to 
generate characters such as "comnriand-option-A" 
or "control-alt-X" using the graphical keyboard. 

Fig. 17 is a series of views showing an exam- 
ple of selection from an 8-item marking menu in an 
embodiment of the present invention. The eight 
menu selections include "uppercase" (shift), "op- 
tion," "Ctrl", "command." "backspace." "space," 
"return," and "delete." 

In the first view of Fig. 17, the character buffer 
contains a character, "A," that the user wants to 
delete. This unwanted character is immediately be- 
hind the cursor position, which is represented by 
the small triangle. The user presses and holds the 
pen against the display screen at the location of a 
key (here, "d") of the keyboard image of the 
graphical keyboard. The key is highlighted in the 
display. Thereafter, a radial menu showing eight 
selections, including four choice: of modifier-"up- 
percase," "option," "control." and "com- 
mand"— and four choices of special characters- 
- "backspace," "space," "return," and (not visible in 
Fig. 17) "delete"— appears on the screen, overlaid 
on the keyboard image and approximately centered 
around the location of the tip of the pen. At this 
point, the user can simply lift the pen off the 
screen to cancel the menu display, in which case 
no character is generated by the graphical key- 
board. 

In the second view of Fig. 17, the user moves 
the pen diagonally upward and to the left, in the 
direction of the "backspace" item of the radial 
menu, while maintaining the pen in contact with the 
screen. This selects the "backspace" character. 
The radial menu remains displayed. The key re- 
mains highlighted. 

In the third view of Fig. 17, the user lifts the 
pen point off the screen. This causes the graphical 
keyboard to generate the "backspace" character, 
which in this embodiment causes the unwanted 
character in the buffer to be erased and the cursor 
position to be moved to the left. (In other embodi- 
ments, the "backspace" character, or an equivalent 
such as "'^h." Is placed In the buffer, and it Is left 
to an applications program receiving input from the 
graphical keyboard to interpret this as an erasure.) 
The highlighted key of the keyboard is ignored. 
The radial menu disappears, as does the key high- 
lighting, leaving the keyboard image as it originally 



appeared. 

Additional examples of marking menus as used 
in the present invention were seen in Figs 7-10. 
However, unlike the marking menus illustrated in 
5 Figs. 15-17, the marking menus of Figs. 7-10 are in 
mark mode, so that no radial menus appear. 

In some embodiments of the invention, marking 
menus are optional. The menus can be optionally 
disabled by the expert user, so that marks can be 
10 used without menus. In some embodiments, the 
marks can even be invisible, as though drawn in 
invisible ink. The path of the pen point (or mouse 
cursor or other input device representation) as 
drawn by the user is interpreted by the graphical 
75 keyboard as though a mark had been drawn, but 
no ink trail appears on the screen. 

5. System Components 

20 Fig. 18 illustrates a system 1 suitable for im- 

plementation of the graphical keyboard of the 
present invention. System 1 includes a processor 
5, memory 6 coupled to processor 5, user interface 
software 7 executed by processor 5, a pen 10, an 

25 input-sensing display device 20 that includes an 
input sensor 21 closely coupled to a display screen 
22, a character buffer 30. and an event driver 40 
that incorporates a delay timer 45. Screen 22 dis- 
plays & keyboard image 25 including a plurality of 

30 keys 26. Sensor 21. screen 22. buffer 30, and 
event driver 40 are operationally connected to pro- 
cessor 5 and are accessible to user interface soft- 
ware 7. System 1 typically further includes operat- 
ing software (not shown) executed by processor 5 

35 and hardware (not shown) for interconnecting var- 
ious components of system 1 . Application program 
50, which is not part of system 1. accepts char- 
acters supplied by system 1 (that is, by the graphi- 
cal keytxiard) via buffer 30. 

40 In use. system 1 receives input in the form of 
taps and strokes from pen 10 applied to input 
sensor 21. Processor 5, carrying out instructions 
according to user interface software 7, determines 
the significance of the pen strokes, and in particu- 

45 lar the characters to which the pen taps and marks 
correspond. Processor 5 outputs a stream of the 
characters thus determined to buffer 30, and op- 
tionally outputs visual feedback, such as Ink trails 
and pop-up menus, to screen 22. The characters 

50 stored in buffer 30 can then sent on for further 
processing by application program 50, which can 
be executed by processor 5 (as shown in Fig. 18) 
or by an additional processor or processors (not 
shown). Application program 50 optionally can 

66 echo the characters as part of its visual display, 
which can also appear in screen 22. 

Certain elements of system 1 will now be more 
fully described. Processor 5 is a computer pro- 

10 
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cessing unit (CPU). Typically it Is part of a main- 
frame, workstation, or personal connputer. It can 
include multiple processing elements In some em- 
bodiments. 

Pen 10 is used in conjunction with sensor 21 to 
provide input to the graphical keyboard. Pen 10 
can be any instrument suitable for use with sensor 
2. For example, If device 20 is a touch-sensitive 
display screen, pen 10 can be a simple stylus of 
plastic, metal, wood, or other material or materials. 
As another example, if device 20 is a light-sensitive 
screen, pen 10 can be a light pen. In some em- 
bodiments, an ordinary pen or pencil, a laser point- 
er, or even the user's finger can serve as pen 10. 

Sensor 21 is used to detect whether pen 10 is 
in contact with device 20 and. if so. to determine 
the position of pen 10 with respect to device 20. 
That is, sensor 21 is used to determine whether 
pen 10 is "up" or "down" and where pen 10 is with 
respect to display screen 22 and in particular with 
respect to keyboard image 25. Sensor 21 substan- 
tially overlays (or, in some embodiments, under- 
lays) screen 22 so that the position of pen 10 
relative to sensor 21 coincides with the position of 
pen 10 relative to screen 22. 

The marking menus technique described 
above presupposes that pen 10 affects the display 
shown by screen 22 in a manner analogous to the 
way an ordinary ball-point or fountain pen affects 
paper. That is, pen 10 leaves an ink trail on screen 
22. Thus when a user makes a stroke or a com- 
bination of strokes with pen 10 on input sensor 21, 
for example, a stroke that proceeds leftward and 
then upward, a corresponding mar/c appears on the 
screen, for example, an L-shaped mark. 

Keyboard image 25 is a graphical image gen- 
erated by processor 5 in accordance with user 
interface software 7 and displayed on screen 22. 
Keyboard image 25 is shown in Fig. 18 as includ- 
ing keys 26 in a standard QWERTY keyboard 
layout. Other kinds of keyboards and layouts can 
be used within the scope of the invention; for 
example, keyboard image 25 can include function 
keys, or can be the image of a nonstandard key- 
board layout such as a Dvorak keyboard. In some 
embodiments, keyboard image 25 can be the im- 
age of a numeric keypad or a musical keyboard. 

Keyboard image 25 as shown In Fig. 18 does 
not include modifier keys such as "shift," "Ctrl," 
"alt," or "command." According to the invention, 
these keys are unnecessary; their functionality is 
replaced in this embodiment by appropriate pen 
strokes. Thus keyboard image 25 occupies a small- 
er area of screen 20 than would be the case were 
the additional keys present. 

Buffer 30 stores characters that are generated 
by the graphical keyboard in response to user 
input. Typically, these characters are accepted as 



input by an application program such as application 
program 50. Screen 22 optionally can display 
("echo") the characters that are stored in buffer 30 
or the characters that application program 50 re- 

5 ceives. Buffer 30 can be implemented using mem- 
ory 6. It is accessible by both user interface soft- 
ware 7 and application program 50. 

Event driver 40 is connected to processor 5, 
input sensor 21 , and delay timer 45, and is acces- 

10 sible by user interface software 7, Typically, event 
driver 40 and delay timer 45 are implemented in 
operating system software that is executed by pro- 
cessor 5. They can also be implemented in hard- 
ware or partially in hardware in some embodi- 

15 ments. 

5.1 Event-driven architecture 

Event driver 40 in used to inform the graphical 
20 keyboard about certain events, or changes of state, 
associated with pen 10. Specifically, event driver 
40 can generate the following events: 

PenDown - Thin event in generated when pen 
10 in pressed down, that is, when pen 10 is newly 
25 brought into contact with sensor 21 . 

PenDrag - This event is generated when pen 
10 is kept pressed down and the location of pen 10 
changes. A PenDrag event in generated every time 
the pen moves an appreciable amount, e.g. one 
30 pixel. 

PenUp - This event is generated when pen 10 
is lifted up from (removed from contact with) sen- 
sor 21 . 

PenStill - This event is generated when pen 

36 1 0 is kept pressed on sensor 21 and the location of 
pen 10 with respect to sensor 21 does not change 
a substantial amount (typically, 2 mm) for E sec- 
onds, where E is a delay time. E can be a fixed 
time value in some embodiments and can be ad- 

40 justed by the user in other embodiments. Typically. 
E is in the range of approximately .3 seconds to 1 
second. Delay timer 45 is used to determine the 
length of time that pen 10 has been kept pressed 
down and held still. Timer 45 signals event driver 

45 40 when the elapsed time reaches E seconds, 
which causes event driver 40 to generate a PenStill 
event. Timer 45 begins counting when pen 10 in 
pressed down or stops moving and in reset when 
pen 10 Is lifted or begins moving. 

50 The four kinds of events are used In various 
combinations to produce different behavior of the 
graphical keyboard of the invention. For example, 
suppose the user wants to enter a modified char- 
acter, such as " ^ a." The user initiates the process 

66 of entering the character by pressing the pen down 
on a key of the graphical keyboard, which causes 
event driver 40 to generate a PenDown event. The 
user then makes a mark by dragging the pen while 
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holding it down. In the process of doing so, the 
user causes event driver 40 to generate numerous 
PenDrag events, Optionally, the user can pause 
during the marking to prompt the system for a 
menu (either a top-level menu or a submenu) to be 
displayed. This causes event driver 40 to generate 
a PenStill event, Finally, to complete the mark, the 
user lifts the pen away from the screen or input 
sensor, which causes event driver 40 to generate a 
PenUp event. Thereafter, the user's mark Is pro- to 
cessed by processor 5 according to the basic 
character determined by the pen location at the 
time of the PenDown event and the modifier(s) 
determined by the mark. This typically causes the 
desired character to be output to character buffer is 
30. 

5.2 Software environment 

As will be appreciated by persons skilled in the 20 
art, the graphical keyboard of the present invention 
is typically used as part of a graphical user inter- 
face in the context of an operating system or 
environment. This environment is supported by 
software that is executed by processor 5. Typically, 25 
the environment supports windowing operations 
such as associating a window with keyboard image 
25, displaying this window at a location on screen 
22, determining whether events associated with 
pen 10 pertain to this window and therefore to the 30 
graphical keyboard, directing position information 
from pen 10 to this window and thence to the 
graphical keyboard, and directing the output of 
character buffer 30 to an application program such 
as application program 50. 35 

Typically, the software environment operates in 
conjunction with event driver 40 to feed to the 
graphical keyboard all and only those events rel- 
evant to operation of the graphical keyboard. 
Events that have to do with other windows or 40 
processes are directed to those other windows or 
processes. In such embodiments the graphical key- 
board need not Incorporate logic to discriminate 
between events that are relevant to its operation 
and events that are not. It is assumed here (in 45 
particular, in section 6 below) that the events pro- 
cessed by the graphical keyboard of the invention 
are relevant events. 

Other kinds of software environments, including 
nonwindowing . environments, are contemplated so 
within the scope of the invention. For example, the 
graphical keyboard of the Invention can be imple- 
mented in a pen-based computer system that 
presents the user interface through the metaphor of 
a continuous sheet or roll of paper rather than that 66 
of a collection of discrete windows. 
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5.3 Alternatives to Input-Sensing Screens 

In the embodiment of Fig. 18, input sensor 21 
is closely coupled to screen 22 in input-sensing 
display device 20. This makes the relative positions 
of pen 10 and the keys 26 of keyboard image 25 
immediately apparent to the user. Such an arrange- 
ment, in which the user effectively writes on the 
display, can be termed a direct interface. The 
invention can also be adapted for use with an 
indirect interface, in which the pen or other writ- 
ing instrument does not contact the display. Some 
examples of such embodiments will now be given. 

In the alternative embodiment shown in Fig. 19, 
sensor 21 and screen 22 are decoupled, so that a 
conventional, non-input-sensing display can be 
used. A digitizing tablet 70 is used to detect input 
from pen 10, and the position of pen 10 on tablet 
70 is represented by the position of an indicator 
such as cursor 27 in screen 22. The user can 
determine the relative position of pen 10 with re- 
spect to keys 26 of keyboard image 25 by inspect- 
ing the position of cursor 27. 

In yet another alternative embodiment, as 
shown in Rg. 20, a mouse 80 (or other equivalent 
input device) takes the place of both pen 10 and 
sensor 21 . The mouse position is sensed by appro- 
priate software that is part of user interface soft- 
ware 7 executed by processor 5. The status of 
mouse button 81 can be used to determine wheth- 
er mouse 80 is active: this is analogous to using 
sensor 21 to determine whether pen 10 is down or 
up. Cursor 27 provides visual feedback about the 
relative positions of mouse 80 and keys 26 of 
keyboard image 25. Persons skilled in the art will 
appreciate that a broad range of equivalent input 
devices sensing two or more dimensions, including 
but not limited to trackballs and joysticks, can be 
used in place of mouse 80. 

In some embodiments, keyboard image 25 can 
be stenciled, glued, or otherwise affixed onto a 
writing surface in addition to or in lieu of being 
displayed on an electronic display screen. The 
writing surface can be. for example, digitizing tablet 
70 of the embodiment of Rg. 19 or a mouse pad 
used in conjunction with mouse 80 of the embodi- 
ment of Fig. 20. Fig. 19A illustrates a digitizing 
tablet 70' to be used in place of digitizing tablet 70 
in the embodiment of Fig. 19. Digitizing tablet 70* 
has stenciled on it a keyboard image 25* that 
Includes keys 26\ The user can interact with the 
graphical keyboard by using pen 10 to make 
strokes across keys 26' of stenciled keyboard Im- 
age 25*. Stenciled keyboard image 25* can have a 
counterpart graphical keyboard image 25 displayed 
on display screen 20. with the position of pen 10 
being indicated by a cursor 27 and the pen stroke 
path indicated by an ink trail (not shown in Fig. 19). 
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Alternatively, the on-screen visual cues (that is, 
cursor 27 and keyboard image 25 including keys 
26) can be onnltted in some embodiments, in which 
case stenciled keyboard image 25* replaces dis- 
play 20 in communicating character information, 
position information, and the like to the user of the 
graphical keyboard of the Invention, In still other 
embodiments, stenciled keyboard image 25' can 
be applied to a dedicated portion of an input- 
sensing display screen or can be applied to a 
suitably constructed membrane input sensor. 

6. Method Steps 

Figs. 21-28 are flowcharts of the steps involved 
in processing Input from the graphical keyboard of 
the invention in an embodiment that incorporates 
marking menus. These steps are carried out by 
system 1 under the control of processor 5, in 
response to PenDown, PenDrag, PenStill. and 
PenUp events generated by event driver 40. 

6.1 Data structures 

The event handling routines that will be de- 
scribed share certain data structures in common. 
These data structures are typically stored in mem- 
ory 6 and are accessible by processor 5. In one 
embodiment the data structures are of the following 
types: 

(1) MM - represents a menu. An MM includes: 

• a list of item names (e.g., shift, control, 
command, option) 

• a list of submenus (represented as other 
MMs) 

• a selected item, which is an index into the 
list of item names and list of submenus 

Before processing of a character starts (that is, 
before a new PenDown event occurs), a hierar- 
chy of MMs exists which describes the hierar- 
chy of menus that can be displayed. The MM 
hierarchy can be updated to reflect, in menu 
mode, the sequence of menus that has been 
displayed so far since the most recent PenDown 
event, or in mark mode, the sequence of menus 
that is selected by the current mark. 

(2) Point - a tuple (x.y,t.<<»): 

X - a horizontal screen location 
y - a vertical screen location 
t - the time x and y were recorded 
^ - the angle formed by the preceding 
point, this point, and the next 

(3) PolntLlst - an ordered list of Points. A mark 
is represented by a PolntLlst that includes the 
screen (or input sensor) locations of the pen as 
the mark is being drawn. Each location has 
associated with it the time when it occurred 
(e.g., measured in milliseconds from the most 



recent PenDown event). 

(4) ModeFlag - a toggle or flag that indicates 
whether the current mode for marking menus is 
nnenu mode or mark mode. 

5 

6.2 Event handling routines 

Fig. 21 shows the steps for responding to one 
of the four kinds of events generated by event 

10 driver 40: PenDown, PenDrag, PenStill. and PenUp. 
First the type of event that is to be handled is 
determined (Step T). According to the event type, 
one of four different event response routines is 
called: PenDown (Step A), PenDrag (Step B), Pen- 

15 Still (Step C). or PenUp (Step D). This completes 
processing until the next event occurs. 

(1) PenDown. If the event is a PenDown event 
(step A), the routine whose flowchart appears in 
Fig. 22 is executed. A determination is made of 

20 which key of the graphical keyboard is being 

pressed (Step AA). Typically, if an input-sensing 
display screen is used for input, the key being 
pressed is the key within whose borders the tip 
of the pen rests. If a mouse or a tablet separate 

25 from the screen is used for input, the key being 
pressed is the key whose position correlates 
most closely with the display position of the 
cursor 27 that represents the mouse or pen. 
Next, the key being pressed is highlighted in the 

30 display (Step AB). Typically this is done by 
darkening the appearance of its borders or by 
changing the color of its borders. Next, the 
mode is set to mark mode (Step AC). This is 
done by setting the ModeFlag to indicate mark 

35 mode. Finally, the pen location and the current 
time are saved as a Point in a PointList that is 
used to represent the mark currently being 
made (Step AD). The tuple of values of this 
Point indicates the location of pen 10 with re- 

40 spect to screen 22 (or the corresponding posi- 
tion of cursor 27 in embodiments that represent 
pen- position as cursor position). 

(2) PenDrag. If the event is a PenDrag event 
(Step B). the routine whose flowchart appears in 

45 Fig. 23 Is executed. The current mode is deter- 
mined by inspecting the ModeFlag (Step BA). If 
the current mode is mark mode, the pen loca- 
tion and time are saved as a Point in the current 
PointList (Step BB) and an Ink trail Is drawn 

50 from the previous pen location to the current 
pen location (Step BC). This causes a mark to 
appear on the display screen along the path of 
the pen as the pen Is dragged. As the user 
drags the pen to make a mark, typically nu- 

55 merous PenDrag events are generated. 

If the current mode is menu mode, a menu 
is already on the screen. A determination is 
made of which menu item is being selected 
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based on the pen location (Step BD) and this 
selection Is stored in the "selected item" field of 
the MM that represents the current menu (Step 

BE). 

In menu mode, the user can select an item s 
from the current menu by dragging the pen from 
the center circle of the displayed menu into a 
radial sector (i.e., a "pie slice") of the menu. As 
the user drags the pen, a series of PenDrag 
events occurs. The user can continue to move io 
the pen, even outside the menu's perimeter, and 
the selection is maintained so long as the user 
stays within the angular boundaries of the sec- 
tor. Moving the pen into another sector selects a 
different menu item and deselects the previous- is 
ly selected menu item. Thus the menu selection 
just described is a revocable one. The selection 
does not cause the graphical keyboard to gen- 
erate a character or a modifier immediately. The 
actual application of the currently selected menu 20 
item to generate or modify a character occurs 
only on a PenUp event. 

(3) PenStill. If the event Is a PenStill event 
(Step C), the routine whose flowchart appears in 
Rg. 24 is executed. The current selection mode 25 
is determined by inspecting the ModeFlag (Step 

CA) . By holding the pen still, the user intends to 
cause menus to be displayed. Thus if the cur- 
rent mode is mark mode, the mode is switched 

to menu mode by toggling the ModeFlag (Step 30 

CB) , and a routine is called to determine the 
menu selections corresponding to the mark that 
has been drawn so far (Step CC). This routine, 
which is a straightforward extension of the rou- 
tine described below with reference to Fig. 26, 35 
updates the MM hierarchy to reflect the menus 

that have been selected and updates the display 
accordingly. The current menu (which can be 
either a top-level menu or a submenu) Is dis- 
played at the pen point location. Any mark that 40 
has been drawn is replaced in the display with 
one or more straight line segments that repre- 
sent a schematic, cleaned-up version of the 
mark. If the mark includes no substantial 
changes of direction, a single straight line seg- 45 
ment is displayed. The line segment has end- 
points at the origin of the mark and at the pen 
point. If the mark substantially changes direction 
at one or more intermediate points, which typi- 
cally Indicates that the user Intends to apply a so 
compound modifier to the basic character, a zig- 
zag composed of distinct straight line segments 
is displayed. Each line segment of the zig-zag 
corresponds to a portion of the mark that in- 
cludes no substantial changes in direction. 55 

If the current mode is menu mode, a menu 
is already being displayed and the user presum- 
ably intends to select an item from that menu. 
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Accordingly, if the pen is positioned within a 
sector of the displayed menu (Step CD), so as 
to indicate that the user has already selected an 
item from the displayed menu, and if there is a 
submenu corresponding to the indicated menu 
item (Step CE). the current menu is removed 
from the display (Step CP) and the submenu 
corresponding to the indicated menu item is 
displayed and becomes the current menu (Step 
CG). The MM hierarchy is updated accordingly. 
The example of Fig. 16 shows how this appears 
to the user: The second view of Fig, 16 shows a 
main menu being displayed, and the third view 
of Fig. 16 shows the main menu eliminated from 
the display and replaced by a submenu. If the 
pen is not within a sector of the displayed menu 
(Step CD) or there is no submenu correspond- 
ing to the indicated menu item (Step CE), the 
event response routine simply returns, leaving 
the current menu displayed. 
(4) PenUp. If the event is a Pen Up event (Step 
D), the routine whose flowchart appears in Fig. 
25 is executed. The current selection mode Is 
determined by inspecting the ModeFlag (Step 
DA). If the current mode is mark mode, any 
mark that has been drawn is erased from the 
display (Step DB). and the menu selections that 
correspond to the mark are determined accord- 
ing to the routine to be described below with 
reference to Fig. 26 (Step DC). After this deter- 
mination, the MM hierarchy reflects the selected 
menu items. Thus the modifiers or special char- 
acters that correspond to the currently displayed 
menu and any menus previously displayed 
since the most recent PenDown event are re- 
flected in the MM hierarchy. If the mode is 
menu mode, all menus and lines are erased 
from the display (Step DD). The routine for 
determining menu selections is not called in this 
case, because the Mfs/I hierarchy is already up 
to date. 

By lifting the pen off the screen, the user 
Intends to complete a mark and to cause the 
graphical keyboard to generate a character to be 
placed In the keyboard buffer. Accordingly (regard- 
less of the current mode) the MM hierarchy is 
traversed to determine which modifiers or special 
characters were selected (Step DE). Then the ap- 
propriate character is generated according to and 
placed Into character buffer 30 (Step DF). This 
character depends on the user's menu selections, 
which are encoded in the MM hierarchy. It can be 
a modified (or unmodified) version of the basic 
character whose key was pressed at the most 
recent PenDown event. For example, if the basic 
character is "a," and the "Ctrl" and "uppercase" 
modifiers are selected, the output character is 
•""A." Alternatively, if the user's mark or menu 
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selections indicate a special character such as 
backspace or return, that special character is the 
character to be generated, and the pressed key Is 
Ignored. 

Finally, the l^ey that was pressed is unhighligh- 
ted, that Is, restored to its normal appearance In 
the display (Step DG), This completes the cycle of 
events, and the graphical keyboard Is ready to 
accept a new character at the next PenDown event. 

6.3 Determining Modifiers 

Fig. 26 flowcharts the steps of a routine for 
determining what modifiers (or special characters) 
correspond to a given mark drawn by the user. 
This routine is called in Step DC of Fig. 25, as part 
of the processing of a PenUp event. The routine 
accepts as input a mark (e.g.. represented as a 
PointList) and returns as output a set (possibly the 
empty set) of modifiers. 

Initially. the mark is tested to see if it is of 
appreciable length, e.g.. longer than a test value 
such as 5 mm (Step EA). If the mark is shorter 
than the test value, it is considered not to be a 
mark at all, as though the user had simply tapped 
the key. In this case, the empty set is returned 
(Step EB). 

If the mark is as long or longer than the test 
value, the mark is analyzed to determine Its articu- 
lation points (Step EC). This analysis is carried out 
according to the routine that is described below 
with reference to Fig. 27. Thereafter, the articula- 
tion points are matched with the centers of cor- 
responding menus and submenus to determine the 
selected modifiers (Step ED). This analysis is car- 
ried out according to the routine that is described 
below with reference to Rg. 28. According to these 
"effective" or "virtual" menu selections, a set of 
modifiers is generated and returned as output 
(Step EE), 

6.4 Determining Articulation Points 

Generally speaking, the articulation points of 
a mark are the points along the mark at which the 
user Indicates selections from a menu or submenu. 
Typically, this includes the first (starting) and last 
(finishing) points of the mark and any points at 
which the mark undergoes a substantial directional 
change. For example, If the radial menu has eight 
items, so that each sector of the menu occupies 45 
degrees, a change of greater than about 22.5 de- 
grees in angular heading can be considered sub- 
stantial. Substantial angular changes are deter- 
mined after the mark has been filtered for noise 
and jitter. 

More specifically, the articulation points of a 
mark are determined in one embodiment according 



to the routine whose flowchart appears In Fig. 27. 
This routine, which in this embodiment is called 
GetArticulationPolnts. is called in Step EC of Fig. 
26. The routine accepts as input the PointList for 

5 the current mark (the input PointList) and outputs 
another PointList (the output PointList) which con- 
tains only the articulation points. 

At the start of the routine, the input PointList is 
examined to determine whether and where the user 

10 paused significantly while drawing the mark (Step 

FA) . This can be done, for example, by seeking 
pairs of points between which the elapsed time is 
greater than a specified delay time such as 0.5 
seconds. (Typically the value used for the delay 

75 time here is less than the value used for the delay 
time used to trigger a PenStill event.) 

If at least two pause points are found (Step 

FB) . then if the starting point of the mark is not 
included among the pause points, it Is added to the 

20 output PointList (Step FC). The pause points are 
then without further analysis taken to be articulation 
points and are added to the end of the output 
PointList in the order of their appearance in the 
input PointList (Step FD). The finishing point of the 

25 mark is then added to the very end of the output 
PointList (Step FE). This completes the routine. 

If fewer than two pause points are found, the 
input PointList is analyzed further. First, the data of 
the PointList is preprocessed to remove certain 

30 inaccuracies and artifacts. In particular, inaccura- 
cies that occur when the pen first touches the 
display are removed (Step FF). as are inaccuracies 
that occur when the pen is finally lifted from the 
display (Step FG). Also, small angle changes due 

35 to pen jitter are filtered out (Step FH). 

After the PointList has been "cleaned up" in 
this manner, it is searched to detect points at which 
the mark forms a significant angle-that is, points at 
which the direction of the pen's motion changes 

40 sufficiently that a menu selection would be made if 
the menus were being displayed. This can be 
done, for example, by locating Points for which the 
value of <^ exceeds a threshold angle (Step FJ). 
The threshold angle typically depends on the num- 

45 ber of menu choices. For example, if the marking 
menus support 8 choices, as is the case for the 
menus illustrated in the example of Fig. 14, then 
angular changes of greater than about 22.5 de- 
grees can be considered significant and the thresh- 
so old angle can be set to 22.5 degrees. If the menus 
support only 4 choices, as is the case for the 
menus illustrated in the examples of Figs. 15 and 
16. then the threshold angle can be set to 45 
degrees. 

55 If no significant angular changes are detected 

(Step FK), the mark is considered to be a straight 
line, so that the first and last points of the mark are 
placed in the output PointList to be returned as the 

15 
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articulation points (Step FL). Otherwise, if one or 
more significant angular clianges are detected, the 
mark is considered to include an angle or angles, 
so that the first point, last point, and the point or 
points of largest angular change are placed in the 
output PointList to be returned as the articulation 
points (Step FM). This completes the routine. 

6.5 Matching Articulation Points with Corresponding 
Menus 

Generally speal<ing, once the articulation points 
have been found, the mark is interpreted as though 
the system had been in menu mode and the user 
had paused each articulation point, so that the 
appropriate menu and submenu(s) had been dis- 
played at the articulation points of the mark. More 
specifically, in one embodiment the routine whose 
flowchart appears in Fig. 28 is used to interpret the 
articulation points. This routine, which in this em- 
bodiment is called ArticulationPointsToMenultems. 
is called In Step ED of Fig. 26. 

ArticulationPointsToMenultems accepts as in- 
put the PointList that contains the articulation points 
for the current mark, which was returned as the 
output PointList from the GetArticulatronPoints rou- 
tine called in Step EC of Fig. 26. and also the 
hierarchy of MM data stmctures that describes the 
hierarchy of menus and submenus that can be 
displayed. The routine outputs the MM hierarchy 
with the appropriate selected items, wNch repre- 
sents the set of modifiers. 

The routine begins by selecting the topmost 
menu from the MM hierarchy to be the current 
menu (Step GA). Thereafter the routine loops until 
either the hierarchy of menus is exhausted or the 
list of articulation points is exhausted, whichever 
comes first. 

More particularly, the loop proceeds as follows: 
For each' articulation point in the PointList other 
than the last one (Step GB), a corresponding menu 
selection is determined (Step GC). This is the 
menu item that would be selected if the current 
menu were centered on the current articulation 
point and the pen were positioned at the location of 
the next articulation point in the PointList. The 
menu selection is stored as the "selected item" for 
the current menu by making appropriate modifica- 
tions to the contents of the MM data structure that 
represents the current menu (Step GD). Then a 
test is made to determine whether the selected 
menu item has a submenu associated with it. If the 
selected item has no submenu (Step GE). the 
menu hierarchy is exhausted and the routine is 
complete. In this case, the loop terminates Imme- 
diately (Step GF)- Otherwise, the submenu is taken 
as the new current menu (Step GG) and the loop 
repeats until all articulation points other than the 



last one have been processed (Step GH). The 
routine Is then complete. Upon return from this 
routine, the hierarchy of MM data structures re- 
flects the set of selected menu items, and thus the 
5 set of modifiers to be generated and returned In 
Step EE of Rg. 26. 

6.6 Using Event Sequences to Generate Characters 

10 Sequences of events triggered by user actions 
cause the graphical keyboard to generate char- 
acters. For example, if a user wants to enter a 
lowercase character, such as "a," the user taps the 
pen on the desired key of the keyboard image. 

75 This gives rise to the sequence of events (Pen- 
Down, PenUp). This .event sequence, in turn, 
causes the steps associated with these events to 
be performed by system 1 according to the logic of 
the flowchart of Fig. 21. More particularly, upon the 

20 occurrence of the PenDown event, processor 5 
determines which key of keys 26 is pressed, 
causes that key to be highlighted in display screen 
22. sets the mode to marking, and saves the loca- 
tion of pen 10. Thereafter, upon the occurrence of 

25 the PenUp event, processor 5 determines that the 
mode is marking. Because no mark has been 
made and no modifier has been applied, processor 
5 simply unhighllghts the key in display screen 22 
and places the ummodified character correspond- 

30 ing to the key into character buffer 30. 

To enter a modified character, such as "A", 
"^a," or "'^A," without causing menus to be dis- 
played, the user touches the pen to the desired 
key. drags the pen to make the desired mark, and 

35 then lifts the pen. This gives rise to the sequence 
of events (PenDown, PenDrag. PenDrag, PenDrag 
... PenDrag, PenUp) and causes the appropriate 
associated steps to be carried out. More particu- 
larly, upon the occurrence of the PenDown event, 

40 processor 5 determines which key of keys 26 is 
pressed, causes that key to be highlighted in dis- 
play screen 22, sets the mode to marking, and 
saves the location of pen 10. Thereafter, upon the 
occurrences of the PenDrag events, processor 5 

46 determines that the mode is marking, saves the 
cunrent location of pen 10, and updates the ink trail 
displayed on screen 22. Thereafter, upon the oc- 
currence of the PenUp event, processor 5 deter- 
mines that the mode is marking, erases the ink trail 

60 from the display of screen 22, and determines 
which modifier or modifiers have been chosen ac- 
cording to the articulation points in the mark. Also, 
processor 5 unhighllghts the key in display screen 
22 and places the modified character correspond- 

55 Ing to the key into character buffer 30. 

Similarly, to generate a frequently used char- 
acter that is represented by a diagonal mark, such 
as backspace or return, without causing menus to 
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be displayed, the user touches the pen to any key. 
drags the pen to make the desired mark, and then 
lifts the pen. Again this gives rise to the sequence 
of events (PenDown, PenDrag. PenDrag, PenDrag 
... PenDrag, PenUp) and causes the appropriate 
associated steps to be performed. 

To use menus to enter a character having a 
single modifier, such as "A" or "^a," the user 
touches the pen to the desired key and holds it 
there for a sufficient time for a menu to appear 
(e.g., for .3 seconds). The user makes the desired 
menu selection by dragging the pen, and then lifts 
the pen. This gives rise to the sequence of events 
(PenDown. PenStill, PenDrag, PenDrag. PenDrag ... 
PenDrag, PenUp). and the appropriate associated 
steps to be performed. Similarly, to generate a 
frequently used character that is represented by a 
diagonal mark, such as backspace or return, us- 
ing displayed menus, the user touches the pen to 
any key. holds it still until the menu appears, 
makes the desired menu selection by dragging the 
pen. and then lifts the pen. Again this gives rise to 
the sequence of events (PenDown, PenStill, Pen- 
Drag. PenDrag. PenDrag ... PenDrag. PenUp) and 
causes the appropriate associated steps to be car- 
ried out. 

To use menus to enter a character having a 
double modifier, such as ""^A," the user touches 
the pen to the desired key, holds it there until the 
menu appears, makes the desired selection, holds 
the pen still again until the next menu appears, 
makes the desired selection, and lifts the pen. This 
gives rise to the sequence of events (PenDown, 
PenStill, PenDrag, PenDrag. PenDrag ... PenDrag, 
PenStill, PenDrag. PenDrag, PenDrag ... PenDrag, 
PenUp) and causes the appropriate associated 
steps to be performed. Similarly, the user can add 
yet further modifiers using menus by continuing to 
drag and then hold still the pen as needed. Thus to 
use menus to enter a triply modified character, 
such as "command-option-A." the user touches the 
pen to the desired key, holds It there until the 
menu appears, makes the desired selection, holds 
the pen still again until the next menu appears, 
makes the desired selection, holds the pen still yet 
again until the next menu appears, makes the 
desired selection, and lifts the pen. This gives rise 
to the sequence of events (PenDown, PenStill, Pen- 
Drag, PenDrag, PenDrag ... PenDrag, PenStill, Pen- 
Drag, PenDrag, PenDrag ... PenDrag, PenStill. Pen- 
Drag. PenDrag, PenDrag ... PenDrag, PenUp) and 
causes the appropriate associated steps to be per- 
formed. 

The user can also use the mark confirmation 
feature of marking menus to obtain a menu display 
during the process of making a mark representing 
a compound modifier. For example, if the user 
touches the pen to a key. drags the pen to the right 



and then upward (substantially without pausing dur- 
ing the change of direction), and then holds the 
pen still without lifting It away from the screen, 
thereafter the menu for the rightward mark and 

5 submenu for the upward mark are both displayed. 
The sequence of events tor this example is (Pen- 
Down. PenDrag. PenDrag, ... PenDrag, Pendrag. 
PenStill). Once again, the appropriate associated 
steps are carried out. 

10 Still other combinations of events can be sup- 

ported in some embodiments. For example, in 
some embodiments a modifier can be applied to a 
character generated by a diagonal stroke. 

IS 7. Discussion of the Software 

A current embodiment of the invention is im- 
plemented in software on a digital computer. The 
Appendix provides source code for a software pro- 

20 gram for implementation of this embodiment. The 
embodiment supports singly modified characters. 
The source code (unpublished work, copyright 
©1993 Xerox Corporation) includes a single source 
code file. It is written In the HyperCard scripting 

25 language for the Apple Macintosh family of com- 
puters. Persons of skill in the art will appreciate 
that a wide variety of programming languages and 
hardware configurations can readily be used based 
on this disclosure without departing from the scope 

30 of the invention. 

8. Conclusion 

The present invention provides a simple and 

35 easy-to-implement technique that enables com- 
mands and data, such as alphanumeric data, to be 
entered accurately and efficiently into a pen-based 
computer via a graphical keyt>oard. The graphical 
keyboard allows a user to make use of existing 

40 knowledge of keyboard entry. The use of strokes 
and marking menus provides accelerated input. 
Because of the seJf-revealIng nature of marking 
menus, these accelerators are accessible to nov- 
ices. They are also valuable to experienced users 

45 who may need to refamiliarize themselves with the 
menus from time to time, for example, if larger 
menus with a greater variety of commands are 
installed as part of a software upgrade to the 
graphical keyboard. 

50 The graphical keyboard of the invention offers 
improved ease of use over existing graphical key- 
boards. For example, the graphical keyboard of the 
invention is modeless. In previous graphical key- 
boards, tapping a modifier key set a mode (e.g., 

55 shift mode) that explicitly had to be released. This 
inconvenience and awkwardness is avoided in the 
present invention. Moreover, the graphical key- 
board of the invention can be made smaller in 

17 
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terms of screen display size than previous graphi- 
cal keyboards, inasmuch as modifier keys and cer- 
tain command keys such as space and delete need 
not be explicitly represented. Still further, the 
graphical keyboard of the invention allows users to 5 
proceed "by feer-that is. to apply multiple modifi- 
ers and to Input stroke-based commands without 
looking at the displayed keyboard every step of the 
way. Numerous other advantages of the invention 
will be apparent to those of skill in the art. io 

The invention has relevance to most pen-based 
and stylus-based computing systems. In particular, 
the invention lends itself to easy retrofit to existing 
applications. All that is needed for such a retrofit is 
to build a module that the computer system sees is 
as equivalent to the standard mechanical keyboard. 

The invention has been explained with refer- 
ence to specific embodiments. Other embodiments 
are contemplated without departing from the spirit 
and scope of the invention. For example, the basic 
character to be modified by a mark can be des- 
ignated according to the finishing point rather than 
the starting point of the mark. The mark can also 
be decoupled from the key designation; for exam- 
ple, the user can indicate the key with a tap to the 
screen and then draw the mark separately. Dif- 
ferent kinds of strokes, such as double taps (two 
taps of the stylus in rapid succession, analogous to 
the "double click" of a mouse button), can be used 
with the invention. Likewise, pen pressure or ve- 
locity can be used to encode command informa- 
tion. Different kinds of commands can be imple- 
mented, such as font size commands. Alternate 
menu layouts can be used for pop-up menus. 
Menus need not be organized hierarchically. Dif- 
ferent kinds of graphical keyboards and keyboard 
layouts can be used. Moreover, the invention can 
be applied in systems other than those ordinarily 
thought of as pen-based; in particular, it finds ap- 
plication in systems in which a finger, a mouse, a 40 
laser beam, or any one of myriad other instruments 
analogous to a pen is used in lieu of a mechanical 
keyboard to input characters. Still further variations 
and extensions will be apparent to those of skill in 
the art within the scope of the invention. It is 45 
therefore not intended that this invention be limited, 
except as indicated by the appended claims. 

Claims 

50 

1. A user Interface apparatus for a computing 
system, said apparatus comprising: 

a display device displaying a graphical 
keyboard Image; 

an input source comprising a designating 55 
device, said input source providing an Input 
comprising key information and path informa- 
tion, said key information representing a key of 



said graphical keyboard image, said path in- 
formation representing a series of points des- 
ignated by said designating device; and 

an output generator, responsive to said 
path information provided by said input source, 
for producing an output. 

2. The apparatus of claim 1 wherein: 

said graphical keyboard image comprises 
a representation of a plurality of characters; 
and 

said key Information provided by said in- 
put source comprises a character to which said 
key corresponds. 



3. The apparatus of claim 1 wherein said input 
source further comprises a sensor responsive 
to a plurality of positions of said designating 
device, said sensor providing said input source 
20 with position information about said designat- 

ing device during a manipulation of said des- 
ignating device, said position information deter- 
mining said points of said series. 

25 4. The apparatus of claim 3 wherein said sensor 
is integrated with said display device and said 
positions of said plurality include positions in a 
vicinity of said displayed graphical keyboard 
image throughout a substantial portion of said 
30 manipulation of said designating device. 

5. The apparatus of claim 1 wherein said des- 
ignating device comprises a pointing device 
selected from the group consisting of a pen, a 
35 light pen, a stylus, a pencil, a pointer, a 

mouse, a trackball, a joystick, or a finger. 



6. The apparatus of claim 1 wherein said des- 
ignating device comprises a cursor displayed 
by said display device, said cursor providing a 
representation of a position of a plurality of 
positions, a displacement of said cursor deter- 
mining said points of said series. 

7. The apparatus of claim 1 wherein said output 
generator produces said output by applying a 
modifier corresponding to said path information 
to a character corresponding to said key in- 
formation. 

6. The apparatus of claim 7 wherein said modifier 
Is selected from the group consisting of shift, 
uppercase, control, alt. option, and command. 



9. The apparatus of claim 7 wherein said path 
information provided by said input source com- 
prises a direction of displacement of said des- 
ignating device, and wherein said modifier is 
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determined according to said direction of dis- 
placement thus provided. 

10. Tlie apparatus of claim 1 wherein said output 
generator produces said output by determining 
a cliaracter corresponding to said patli infor- 
mation and independently of said key informa- 
tion. 

11. The apparatus of claim 10 wherein said char- 
acter is a special character selected from the 
group consisting of return, space, backspace, 
and delete. 

12. The apparatus of claim 1 further comprising a 
buffer for storing said output. 

13. The apparatus of claim 1 wherein said input 
source provides said key information according 
to a position of said designating device relative 
to said graphical keyboard image. 

14. The apparatus of claim 1 wherein said input 
source provides said key information according 
to one point of said series. 

15. The apparatus of claim 14 wherein said one 
point is a starting point of said series. 

16. The apparatus of claim 1 wherein said display 
device further displays a representation of said 
path information. 

17. The apparatus of claim 16 wherein said repre- 
sentation of said path information comprises a 
mark corresponding to said path information, 
said mark being displayed in conjunction with 
said graphical keyboard Image. 

18. The apparatus of claim 16 wherein said repre- 
sentation of said path information comprises an 
ink trail corresponding to said series of points 
displayed in the vicinity of said graphical key- 
board image. 

19. The apparatus of claim 16 wherein said repre- 
sentation of said path information comprises a 
menu displayed in conjunction with said 
graphical keyboard image. 

20. In a computing system, a graphical keyboard 
apparatus comprising: 

a sensor, responsive to a plurality of posi- 
tions of a position designator relative to said 
sensor; 

a display; and 

a processor, coupled to said sensor and 
said display, said processor: 



displaying, in conjunction with said display, 
a graphical Image of a keyboard, said image 
comprising a plurality of keys, each such key 
being associated with a location in said image; 

5 receiving from said sensor an input com- 

prising position information and path informa- 
tion, said position information comprising a se- 
lected position of said plurality of positions of 
said position designator, said path information 

10 comprising a series of positions of said plural- 

ity of positions of said position designator, said 
series representing a relative displacement of 
said position designator relative to said sensor; 
conelating said selected position with a 

IS location in said image to determine a selected 

location; 

associating said selected location with a 
key of said plurality of keys to determine a 
selected key; 

20 interpreting said path information to deter- 

mine a command; and 

generating an output, said output being 
selected from the group consisting of a char- 
acter determined by applying said command 

25 to said selected key and a character deter- 

mined by executing said command indepen- 
dently of said selected key. 

21. The apparatus of claim 20 further comprising a 
30 buffer coupled to said processor for receiving 

said output thus generated. 

22. The apparatus of claim 20 further comprising 
an event driver, coupled to said sensor and 

35 said processor, for generating events respon- 

sively to changes in said relative position of 
said position designator with respect to said 
sensor, to substantial changes in a proximity 
between said position designator and said sen- 

40 sor. and to substantial pauses in a relative 

motion of said position designator with respect 
to said sensor; and 

wherein said processor generates said out- 
put responsively to an event generated by said 

45 event driver. 

23. The apparatus of claim 20 wherein said pro- 
cessor in conjunction with said display dis- 
plays a representation of said path information 

50 in conjunction with said graphical keyboard 

image. 

24. An apparatus for a graphical keyboard com- 
prising: 

55 a substantially planar display screen dis- 

playing a graphical keyboard image compris- 
ing a plurality of keys; 

a pen means for designating a plurality of 
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38 



positions relative to said graphical keyboard 
image; 

a sensor, coupled to said display, for sens- 
ing positions designated by said pen means; 

a buffer for receiving an output; and s 
a processor, coupled to said display 
screen, said sensor, and said buffer, for gen- 
erating a character responslvely to a series of 
positions designated by said pen means, said 
series of positions describing a relative motion io 
of said pen means with respect to said graphi- 
cal keyboard image, said relative motion com- 
prising a displacement of said pen means sub- 
stantially in a surface corresponding to the 
plane of said display screen. is 

25. The apparatus of claim 24 wherein: 

said display screen and said sensor are 
physically coupled in an input-sensing display; 
and 20 

said surface is substantially planar and is 
substantially parallel to said plane of said dis- 
play screen. 



26. The apparatus of claim 25 wherein said sensor 
overlies said display screen. 

27. The apparatus of claim 25 wherein said sensor 
underlies said display screen. 



25 



30 



28- In a connputing system, a method for using a 
graphical keyboard comprising the steps of: 

displaying a graphical keyboard image via 
a display; 

using an input source comprising a des- 35 
Ignating device to provide an input comprising 
key information and path information, said key 
information representing a key of said graphi- 
cal keyboard image, said path information re- 
presenting a series of points designated by 40 
said designating device; and 

using an output generator to produce an 
output responsivety to said path information. 

29. The method of claim 28 wherein said step of 45 
using said output generator to produce an out- 
put responslvely to said path information com- 
prises the steps of: 

using said output generator to examine 
said series of points to determine a set of 5o 
articulation points; and 

using said output generator to determine a 
command according to said set of articulation 
points. 

55 

30. In a computing system comprising a proces- 
sor, a sensor, a display, and a buffer, a meth- 
od for using a graphical keyboard to produce a 



character, said method comprising the steps 

of: 

displaying on said display a graphical key- 
board Image; 

sensing with said sensor a motion of a pen 
device relative to said sensor, said pen device 
being maintained proximate to said sensor 
throughout a substantial portion of said motion; 

using said processor to generate a char- 
acter responsivefy to said motion; and 

receiving in said buffer said character thus 
generated. 

31- The method of claim 30 wherein said step of 
using said processor to generate said char- 
acter comprises the steps of: 

determining an unmodified character ac- 
cording to an initial position of said pen device 
at a starting point of said motion; 

determining a modifier according. to a path 
taken by said pen device during said motion; 
and 

applying said modifier to said unmodified 
character. 

32. The method of claim 30 wherein said step of 
using said processor to generate said char- 
acter comprises the step of: 

determining a special character according 
to a path taken by said pen device during said 
motion and independently of an initial position 
of said pen device at a starting point of said 
motion. 

33. In a computing system comprising a proces- 
sor, a display, and a designator, a method for 
using a graphical keyboard in conjunction with 
menus, said method comprising the steps of: 

displaying a graphical image of a key- 
board, said keyboard image comprising a re- 
presentation of a plurality of keys, via a dis- 
play; 

displaying a menu, said menu comprising 
a representation of a plurality of items, via said 

display; 

using said designator to designate a key 
from said plurality of keys; . 

using said designator to designate an item 
from said menu; and 

using said processor to generate a char- 
acter according to said designated menu Item. 

34. The method of claim 33 wherein: 

said step of displaying a menu comprises 
displaying a radial menu comprising a repre- 
sentation of a set of modifiers; 

said step of using said designator to des- 
ignate an item comprises selecting a modifier 
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from said set of modifiers; and 

said step of using said processor to gen- 
erate said character comprises applying said 
selected modifier to a ciiaracter corresponding 
to said designated key. 

35. The method of claim 33 wherein: 

said step of displaying a menu comprises 
displaying a radial menu comprising a repre- 
sentation of a set of special characters; 

said step of using said designator to des- 
ignate an item comprises selecting a special 
character from said set of special characters; 
and 

said step of using said processor to gen- 
erate said character comprises producing said 
selected special character independently of 
said designated key. 

36. A user interface apparatus for a computing 
system, said apparatus comprising: 

an input source comprising a designating 
device and a sensor, said input source provid- 
ing an input comprising key information and 
path information, said path information repre- 
senting a series of points designated by said 
designating device: 

means for providing a keyboard image, 
said keyboard image being affixed to a portion 
of said sensor, said keyboard Image compris- 
ing a plurality of characters; 

an output generator, responsive to said 
path information provided by said input source, 
for producing an output. 

37. The apparatus of claim 36 wherein: 

said keyboard image comprises a repre- 
sentation of a plurality of characters; and 

said key information provided by said in- 
put source comprises a character to which said 
key corresponds. 

38. The apparatus of claim 36 wherein: 

said output generator produces said output 
by applying a modifier corresponding to said 
path information to a character corresponding 
to said key information. 

39. The apparatus of claim 36 wherein said output 
generator produces said output by determining 
a character corresponding to said path infor- 
mation and independently of said key informa- 
tion. 

40. In a pen-based computing system comprising 
a processor, a pen, an input-sensing display 
comprising a display screen juxtaposed with a 
sensor, an event driver, and a character buffer , 



a method of using a graphical keyboard in 
conjunction with marking menus, the method 
comprising the steps of: 

displaying a graphical keyboard image on 
5 said display screen, said graphical keyboard 

image comprising a representation of a plural- 
ity of characters; 

using said event driver to generate a se- 
quence of events responsively to a pen stroke 
70 in a vicinity of said displayed graphical key- 

board image, said pen stroke being sensed by 
said sensor, said pen stroke being selected 
from the group consisting of: 

a momentary tap of said pen to said dis- 
75 play screen; 

a pen motion in a leftward direction with 
respect to said display screen; 

a pen motion in a rightward direction with 
respect to said display screen; 
20 a pen motion in an upward direction with 

respect to said display screen; 

a pen motion in a downward direction with 
respect to said display screen; 

a pen motion in a diagonal direction with 
25 respect to said display screen: 

an articulated pen motion comprising a 
combination of pen motions selected from the 
group consisting of leftward, rightward. upward, 
and downward pen motions with respect td 
30 said display screen; and 

a pen motion including a stationary pause 
of duration longer than a delay time; 

responsively to said sequence of events 
thus generated, if said sequence of events 
35 includes an event produced in response to a 

pen motion including a stationary pause of a 
duration longer than a delay time, displaying a 
radial menu on said display screen in a vicinity 
of a pen location associated with said station- 
40 ary pause; 

responsively to said sequence of events 
thus generated, using said processor to gen- 
erate a character and a modifier and to apply 
said modifier to said character, said modifier 
45 selected from the group consisting of: 

no modifier; 

shift; 

uppercase; 
control; 
50 alt; 

option; 
command; 

a combination of at least two modifiers 
selected from the group consisting of shift, 
55 uppercase, control, alt, option, and command; 

and 

substitution of a special character selected 
from the group consisting of return, delete. 
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backspace, and space; 
and 

buffering in said character buffer said 
modified character thus generated. 
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